Pre-trained transformer-based language models for Sundanese

Abstract The Sundanese language has over 32 million speakers worldwide, but the reaped little to no benefits from recent advances in natural understanding. Like other low-resource languages, only alternative is fine-tune existing multilingual models. In this paper, we pre-trained three monolingual Transformer-based models on data. When evaluated a downstream text classification task, found that most of our outperformed larger despite smaller overall pre-training subsequent analyses, benefited strongly corpus size and do not exhibit socially biased behavior. We released for researchers practitioners use.